Course 3 - Advance Computer Vision/W1/ungraded_labs/C3_W1_Lab_2_Transfer_Learning_CIFAR_10.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "private_outputs": true,
      "provenance": [],
      "machine_shape": "hm",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.4"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Imokut/Tensorflow-Advanced-Techniques/blob/main/Course%203%20-%20Advance%20Computer%20Vision/W1/ungraded_labs/C3_W1_Lab_2_Transfer_Learning_CIFAR_10.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FStp_vbUkRz5"
      },
      "source": [
        "\n",
        "\n",
        "\n",
        "# Transfer Learning\n",
        "In this notebook, you will perform transfer learning to train CIFAR-10 dataset on ResNet50 model available in Keras.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qpiJj8ym0v0-"
      },
      "source": [
        "## Imports"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AoilhmYe1b5t"
      },
      "source": [
        "import os, re, time, json\n",
        "import PIL.Image, PIL.ImageFont, PIL.ImageDraw\n",
        "import numpy as np\n",
        "try:\n",
        "  # %tensorflow_version only exists in Colab.\n",
        "  %tensorflow_version 2.x\n",
        "except Exception:\n",
        "  pass\n",
        "import tensorflow as tf\n",
        "from tensorflow.keras.applications.resnet50 import ResNet50\n",
        "from matplotlib import pyplot as plt\n",
        "import tensorflow_datasets as tfds\n",
        "\n",
        "print(\"Tensorflow version \" + tf.__version__)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HuG_q_1jkaZ6"
      },
      "source": [
        "## Parameters"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v4ocPhg6J_xw"
      },
      "source": [
        "- Define the batch size\n",
        "- Define the class (category) names"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cCpkS9C_H7Tl"
      },
      "source": [
        "BATCH_SIZE = 32\n",
        "classes = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O-o96NnyJ_xx"
      },
      "source": [
        "Define some functions that will help you to create some visualizations. (These will be used later)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CfFqJxrzoj5Q"
      },
      "source": [
        "#@title Visualization Utilities[RUN ME]\n",
        "#Matplotlib config\n",
        "plt.rc('image', cmap='gray')\n",
        "plt.rc('grid', linewidth=0)\n",
        "plt.rc('xtick', top=False, bottom=False, labelsize='large')\n",
        "plt.rc('ytick', left=False, right=False, labelsize='large')\n",
        "plt.rc('axes', facecolor='F8F8F8', titlesize=\"large\", edgecolor='white')\n",
        "plt.rc('text', color='a8151a')\n",
        "plt.rc('figure', facecolor='F0F0F0')# Matplotlib fonts\n",
        "MATPLOTLIB_FONT_DIR = os.path.join(os.path.dirname(plt.__file__), \"mpl-data/fonts/ttf\")\n",
        "# utility to display a row of digits with their predictions\n",
        "def display_images(digits, predictions, labels, title):\n",
        "\n",
        "  n = 10\n",
        "\n",
        "  indexes = np.random.choice(len(predictions), size=n)\n",
        "  n_digits = digits[indexes]\n",
        "  n_predictions = predictions[indexes]\n",
        "  n_predictions = n_predictions.reshape((n,))\n",
        "  n_labels = labels[indexes]\n",
        "\n",
        "  fig = plt.figure(figsize=(20, 4))\n",
        "  plt.title(title)\n",
        "  plt.yticks([])\n",
        "  plt.xticks([])\n",
        "\n",
        "  for i in range(10):\n",
        "    ax = fig.add_subplot(1, 10, i+1)\n",
        "    class_index = n_predictions[i]\n",
        "\n",
        "    plt.xlabel(classes[class_index])\n",
        "    plt.xticks([])\n",
        "    plt.yticks([])\n",
        "    plt.imshow(n_digits[i])\n",
        "\n",
        "# utility to display training and validation curves\n",
        "def plot_metrics(metric_name, title, ylim=5):\n",
        "  plt.title(title)\n",
        "  plt.ylim(0,ylim)\n",
        "  plt.plot(history.history[metric_name],color='blue',label=metric_name)\n",
        "  plt.plot(history.history['val_' + metric_name],color='green',label='val_' + metric_name)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wPq4Sw5akosT"
      },
      "source": [
        "## Loading and Preprocessing Data\n",
        "[CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar.html) dataset has 32 x 32 RGB images belonging to 10 classes. You will load the dataset from Keras."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E103YDdQ8NNq"
      },
      "source": [
        "(training_images, training_labels) , (validation_images, validation_labels) = tf.keras.datasets.cifar10.load_data()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "prd944ThNavt"
      },
      "source": [
        "### Visualize Dataset\n",
        "\n",
        "Use the `display_image` to view some of the images and their class labels."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UiokWTuKo88c"
      },
      "source": [
        "display_images(training_images, training_labels, training_labels, \"Training Data\" )"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-q35q41KNfxH"
      },
      "source": [
        "display_images(validation_images, validation_labels, validation_labels, \"Training Data\" )"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "validation_images[0].astype('float32').shape"
      ],
      "metadata": {
        "id": "6iIw-0ousl1C"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ltKfwrCVNuIu"
      },
      "source": [
        "### Preprocess Dataset\n",
        "Here, you'll perform normalization on images in training and validation set.\n",
        "- You'll use the function [preprocess_input](https://github.com/keras-team/keras-applications/blob/master/keras_applications/resnet50.py) from the ResNet50 model in Keras."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JIxdiJVKArC6"
      },
      "source": [
        "def preprocess_image_input(input_images):\n",
        "  input_images = input_images.astype('float32')\n",
        "  output_ims = tf.keras.applications.resnet50.preprocess_input(input_images)\n",
        "  return output_ims\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QOqjKzgAEU-Z"
      },
      "source": [
        "train_X = preprocess_image_input(training_images)\n",
        "valid_X = preprocess_image_input(validation_images)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2fooPL9Gkuox"
      },
      "source": [
        "## Define the Network\n",
        "You will be performing transfer learning on **ResNet50** available in Keras.\n",
        "- You'll load pre-trained **imagenet weights** to the model.\n",
        "- You'll choose to retain all layers of **ResNet50** along with the final classification layers."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "56y8UNFQIVwj"
      },
      "source": [
        "'''\n",
        "Feature Extraction is performed by ResNet50 pretrained on imagenet weights.\n",
        "Input size is 224 x 224.\n",
        "'''\n",
        "def feature_extractor(inputs):\n",
        "\n",
        "  feature_extractor = tf.keras.applications.resnet.ResNet50(input_shape=(224, 224, 3),\n",
        "                                               include_top=False,\n",
        "                                               weights='imagenet')(inputs)\n",
        "  return feature_extractor\n",
        "\n",
        "\n",
        "'''\n",
        "Defines final dense layers and subsequent softmax layer for classification.\n",
        "'''\n",
        "def classifier(inputs):\n",
        "    x = tf.keras.layers.GlobalAveragePooling2D()(inputs)\n",
        "    x = tf.keras.layers.Flatten()(x)\n",
        "    x = tf.keras.layers.Dense(1024, activation=\"relu\")(x)\n",
        "    x = tf.keras.layers.Dense(512, activation=\"relu\")(x)\n",
        "    x = tf.keras.layers.Dense(10, activation=\"softmax\", name=\"classification\")(x)\n",
        "    return x\n",
        "\n",
        "'''\n",
        "Since input image size is (32 x 32), first upsample the image by factor of (7x7) to transform it to (224 x 224)\n",
        "Connect the feature extraction and \"classifier\" layers to build the model.\n",
        "'''\n",
        "def final_model(inputs):\n",
        "\n",
        "    resize = tf.keras.layers.UpSampling2D(size=(7,7))(inputs)\n",
        "\n",
        "    resnet_feature_extractor = feature_extractor(resize)\n",
        "    classification_output = classifier(resnet_feature_extractor)\n",
        "\n",
        "    return classification_output\n",
        "\n",
        "'''\n",
        "Define the model and compile it.\n",
        "Use Stochastic Gradient Descent as the optimizer.\n",
        "Use Sparse Categorical CrossEntropy as the loss function.\n",
        "'''\n",
        "def define_compile_model():\n",
        "  inputs = tf.keras.layers.Input(shape=(32,32,3))\n",
        "\n",
        "  classification_output = final_model(inputs)\n",
        "  model = tf.keras.Model(inputs=inputs, outputs = classification_output)\n",
        "\n",
        "  model.compile(optimizer='SGD',\n",
        "                loss='sparse_categorical_crossentropy',\n",
        "                metrics = ['accuracy'])\n",
        "\n",
        "  return model\n",
        "\n",
        "\n",
        "model = define_compile_model()\n",
        "\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CuhDh8ao8VyB"
      },
      "source": [
        "## Train the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2K6RNDqtJ_xx"
      },
      "source": [
        "# this will take around 20 minutes to complete\n",
        "EPOCHS = 4\n",
        "history = model.fit(train_X, training_labels, epochs=EPOCHS, validation_data = (valid_X, validation_labels), batch_size=64)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CYb5sAEmk4ut"
      },
      "source": [
        "## Evaluate the Model\n",
        "\n",
        "Calculate the loss and accuracy metrics using the model's `.evaluate` function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "io7Fuu-w3PZi"
      },
      "source": [
        "loss, accuracy = model.evaluate(valid_X, validation_labels, batch_size=64)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yml-phRfPeOj"
      },
      "source": [
        "### Plot Loss and Accuracy Curves\n",
        "\n",
        "Plot the loss (in blue) and validation loss (in green)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b1ZMMJ6T921A"
      },
      "source": [
        "plot_metrics(\"loss\", \"Loss\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QbnWIbeJJ_xx"
      },
      "source": [
        "Plot the training accuracy (blue) as well as the validation accuracy (green)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P0YpFs3J99eO"
      },
      "source": [
        "plot_metrics(\"accuracy\", \"Accuracy\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9jFVovcUUVs1"
      },
      "source": [
        "### Visualize predictions\n",
        "You can take a look at the predictions on the validation set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NIQAqkMV9adq"
      },
      "source": [
        "probabilities = model.predict(valid_X, batch_size=64)\n",
        "probabilities = np.argmax(probabilities, axis = 1)\n",
        "\n",
        "display_images(validation_images, probabilities, validation_labels, \"Bad predictions indicated in red.\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "hCuGf1eP2u3K"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}